Multileaving for Online Evaluation of Rankers
نویسنده
چکیده
In online learning to rank we are faced with a tradeoff between exploring new, potentially superior rankers, and exploiting our preexisting knowledge of what rankers have performed well in the past. Multileaving methods offer an attractive approach to this problem since they can efficiently use online feedback to simultaneously evaluate a potentially arbitrary number of rankers. In this talk we discuss some of the main challenges in multileaving, and discuss promising areas for future research. ACM Reference format: Brian Brost. 2017. Multileaving for online evaluation of rankers. In Proceedings of the first Internationl Workshop on LEARning Next gEneration Rankers, Amsterdam, The Netherlands, October 1, 2017 (LEARNER’17), 2
منابع مشابه
Search Engines that Learn from Their Users
More than half the world’s population uses web search engines, resulting in over half a billion queries every single day. For many people, web search engines such as Baidu, Bing, Google, and Yandex are among the first resources they go to when a question arises. Moreover, for many search engines have become the most trusted route to information, more so even than traditional media such as newsp...
متن کاملOverview of the NTCIR-13 OpenLiveQ Task
This is an overview of the NTCIR-13 OpenLiveQ task. This task aims to provide an open live test environment of Yahoo Japan Corporation’s community question-answering service (Yahoo! Chiebukuro) for question retrieval systems. The task was simply defined as follows: given a query and a set of questions with their answers, return a ranked list of questions. Submitted runs were evaluated both offl...
متن کاملDon’t Hurt Them: Learning to Rank from Historical Interaction Data (Keynote)
One of the main advantages of online evaluation schemes is that they are user-based and, as a result, often assumed to give us more realistic insights into the real system quality than off-line methods. This is also one of their main disadvantages: comparing two rankers online requires presenting users with result lists based on those rankers and observing how users interact with them. New rank...
متن کاملInformation vs. Robustness in Rank Aggregation: Models, Algorithms and a Statistical Framework for Evaluation
The rank aggregation problem has been studied extensively in recent years with a focus on how to combine several different rankers to obtain a consensus aggregate ranker. We study the rank aggregation problem from a different perspective: how the individual input rankers impact the performance of the aggregate ranker. We develop a general statistical framework based on a model of how the indivi...
متن کاملAdapting Rankers Online
At the heart of many effective approaches to the core information retrieval problem— identifying relevant content—lies the following three-fold strategy: obtaining contentbased matches, inferring additional ranking criteria and constraints, and combining all of the above so as to arrive at a single ranking of retrieval units. Over the years, many models have been proposed for content-based matc...
متن کامل